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Summarized are evaluation activities and results for 
the first year of an accountability study for trainable mentally 
retarded (TMR) children and youth carried out by the Diival Co^anty 
School Board of Jacksonville^ Florida. It is explained that the 
evaluation design included the construction and implementation of 
criterion measures, statistical analysis of resulting data# 
development of a management information system^ and development of a 
suitable cost-benefit model. It is reported that tests (administered 
three times during the school year) dealt with 398 formulated 
objectives in three areas (social adeguacy, language, vocational 
readiness) at six levels (from Primary I to pre occupational). 
Provided are statistics on percentages of pupils by sex and race, IQ 
levels of pupils, and attendance of pupils. Findings indicate that 
language objectives were generally the most difficult for pupils to 
master; that Primary I objectives had the highest achievement rates; 
that low correlations were found with pupil achievement of objectives 
and school, sex,* race, physical disabilities, enrollment, attendance, 
and teacher; and that strong correlations existed between total score 
and year within level, and between total score and most recent IQ. 
Overall results of the cost-effectiveness analysis are said to have 
shown an average cost of $96 per objective achieved per student. 
Recommendations included the continued development of the 
accountability model and its application to other TMR programs. 
(DB) 
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I OR POLICY 



INTRODUCTION 



The purpose of this report is to provide a 
sugary of the project evaluation activities and their 
results for the first year of an "Accountability Study 
of the Program £or Trainable Mentally Retarded Children 
and Youth". 

The Accountability Study is being carried out by 
the Duval County School Board (DCSB) , Jacksonville, Florida. 
Financial support was obtained under Title VI of the 
Elementary and Secondary Education Act. 

The Institute for Development of Educational Auditing 
(the Institute) has prepared this summary report as a 
part of our program evaluation activities. 

For additional information concerning this program, 
the reader is referred to Mr. William L. Geiger, Project 
Coordinator, or Mr. Keith Brown, Project Evaluator, Duval 
County School Board, Jacksonville, Florida, 
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BACKGROUND 



In -March, 1971, the- Duval County School Board submitted 
a proposal for Federal assistance under Title VI of the Ele- 
mentary and Secondary Education Act to conduct an Accounta- 
bility Study of the Duval County Program for Trainable 
Mentally Retarded Children and Youth, 

The project subsequently was funded, with an overall 
stated project objective "To develop, implement and audit an 
evaluational design based on systems analysis, which will 
measure the cost effectiveness of the Duval County Program 
for Trainable Mentally Retarded Children and Youth in achiev- 
ing' its stated goals in the areas of Language Development, 
Social Adequacy and Vocational Readiness." 

There are over 200 children enrolled in the Program ^ 
grouped into six levels: Primary I, Primary II, Intermediate 
I, Intermediate II, Family Living and Pre-Occupational Train- 
ing. 

The existing curriculum of the pr«:)gram had been 
established from sequentially designed task skills which led 
to the accomplishment of stated program goals for each level. 
These task skills and program goals were refined into 
acceptable behavioral objectives stating minimum levels of 
success as a part of the Accountability Project. 
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Based on performance objectives developed by the Program 
staff, specifications for the "Evaluation Design and Implemen- 
tation" and an "Audit Component" were written and submitted 
for bid in accordance with Duval County and Consolidated City 
of Jacksonville procedures. 

The Institute for Development of Educational Auditing 
was selected as the Evaluation Design and Implementation 
Contractor and the Planar Corporation was selected as the 
Auditor. 



The Institute was contracted to perform the following 
major functions: 

• Construct and implement the criterion measures, 
including pre, post, and interim testing, based 
directly on the Performance Objectives. 

• Conduct necessary statistical analysis of the 
data, to include a method for determining the 
interrelationships of the interim and terminal 
Performance Objectives. 

• Design, develop and implement a management 
information system which will report student 
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progress toward the objectives ana will report 
program costs to be used for cost-benefit analysis. 

• Design, develop and implement a cost-benefit model 
which will provide for analysis of the program and 
its success in developing the maximum potential o f 
trainable children in Duval County, and provide a 
basis for pro gran modi fi cat ion if warranted , in 
meeting the Performance Obiectives. 

• Prepare and submit appropriate reports presenting 
the results. 

Be::ause of the immediate requirement for a "pre-test" 
based on the performance objectives, the design of the test 
construction and administration procedures was the first 
task undertaken by the Institute. 

The first test session was held during the first half of 
December. Eased upon the results, a number of design decisions 
Were made. It was found that questions regarding effects of such 
pupil variables as IQ, age, physical handicaps, sex, and past 
training history could not be answered with data from the first 
administration. The same situation applied to such variables 
as teacher experience, education and attitude; school facilities; 
program level and subject area. Therefore, it was decided to 
structure the design of the second test administration to insure 
a more comprehensive data base of performance measures and to pro- 
vide answers to such evaluation questions as the following: 

• To wh.it extent arf^ the TMR program Objectives, as defined, 
'oin^: ...'hic^cd by ihc tarqct pupiaS, ar> measured by the 




results of the criterion-referenced tcstinq? 



• To what extent is PUPil performance on the Program 
Objectives related to pupil factors including sex, 
age, IQ , physical handicaps, training history, attend- 
ance and family-related factors, as analyzed by appro- 
priate statistical techniques? 

• To what extent is pupil performance on the Program 
Objectives related to instructional factors including 
type of facility, program level, year within program, 
subject area and teacher-related factors, as analyzed 
by appropriate statistical techniques? 

• What are the various costs involved in providing train- 
ing to target Pupils at each proaram level within each 
facility, as measured by the application of a preliminary 
cost-effectiveness model? 

• What modifications in the instructional program will 
occur as the result of pupil performance "feedback" 
to 'Program teachers? 

The second test administration was held during February 
and '^arch and provided much of the performance data needed to 
answer the principal evaluation questions. The third test 
administration was held at the end of the school year and 
represented the final measure of pupil performance for the 
first year of the project. 
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A number of statistical analyses were carried out to 
obtain a "picture" of the objective structure relative to the 
pupils' performance and to estimate objective validity, 
reliability and degree of difficulty. 

To provide an indication of the attitudes of teachers and 
parents tov^ard the Protjram, the Institute developed survey instru- 
ments which were mailed to teachers and parents in June. Both 
instruments contained statements to which respondents indicated 
their opinions on scales ranging from "strongly agree" to 
"strongly disagree". In addition, both parents and teachers 
had the opportunity to list likes and dislikes regarding the 
TMR Program. 

The overall objective for th^;* ^irst nrrijpr-i- vF>ar nallp»d 
for a determination of the cost-effectiveness of the TMR Program 
in achieving its goals. 

There are many approaches to and models for cost-effectiveness 
analysis. The model developed for this program is based upon 
the time and budget constraints for the first project year and 
upon the nature of the cost and performance data available from 
DCSB records. 

Both direct and ind irect program cos ts are determined or 
estim ited from DCSB records. Cost allocation levels are 
identified as system, program, school center and classroom. 
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It was decided that Program effectiveness would be 
measured in terms of numbers of objectives mastered by Program 
pupils during the project year. 

It is possible to assign each objective a value based on its 
relative importance. Such a value could be determined through 
teacher, parent and community involvement. For the present 
measurement, each objective was assigned a value of one. Overall 
Program attainment then is measured by the total estimated 
value points gained during the project year. 

SUMMARY OF EVALUATION FINDINGS 

In carrying out the evaluation activities for the first 
project year, it was essential to collect and maintain a great 
deal of Program data which either was required for immediate 
analysis, was needed for future purposes or was information 
with a high "expected value" which would be lost if not eaptufeu 
as it was generated. 

A total of 398 objectives were written*; and distributed 
among the six program levelis by the Program staff, as follows: 

Program Level Program Area 

Social Vocational 
Adequacy Language Readiness Totals 



Pre Occupational/Family Living 


24 


29 


12 


65 


Pre Occupational (boys only) 


0 


0 


18 


18 


Family Living (girls only) 


0 


0 


31 


31 


Intermediate II 


25 


29 


32 


86 


Intermediate I 


24 


23 


28 


75 


Primary II 


24 


16 


28 


68 


Primary I 


24 


14 


17 


55 


Totals 


121 


111 


166 


398 
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The numbers of Prograin pupils, by Center 
and by Program Level, in the first project 
year data base are as follows: 

Center 



Program Level 


#170 


#201 


TOTAL 


Pre Occupational?' 




1 1 


A C 

4b 


Family Living 


28 


10 


38 


Intermediate II 


35 


13 


48 


Intermediate I 


20 


10 


30 


Primary II 


32 


8 


40 


Primary I 


2 


2 


4 


Reinforcer.ent and 


18 




22 


Adaptive Behavior* 


169 


58 


227 


* The "Reinfoi'd'6inent 


Program" 


and " 


Adaptive 


Behavior'* levels are classes 


for 


students 



who require special treatment. 



The percentages of Program pupils, by sex, 
for each Program Level are shown in the following 
graph. 



Program Level 



Sex 



Pre Occupational/ 
Family Living 

Intermediate II 

Intermediate I 

Primary II 

Primary I 

Reinforcement and 
Adaptive Behavior 



Boys 54% 


Girls 46% 


Boys 52% 


Girls 48% 


Boys 57% 


Girls 43% 


Boys 58% 


Girls 42% 


Boys 75% ""ill' 


Boys 64% 


Girls 36% 



The percentages of Program pupils, by race, 
for each Program Level are as follows: 



Program Level 



Race 



Pre Occupational/ 
Family Living 

Intermediate II 

Intermediate I 

Primary II 

Primary I 

Reinforcement and 
Adaptive Behavior 



White 48% Black 52% 


White 62% 


Black 38% 


White 5 3% 


Black 47% 


White 55% 


Black 45% 


White 100% 


White 62% 


Black 38% 



_ 8 



The minimum, average and maximum 10 's of 
Program pupils, by Program Level, are shown below: 

IQ 

Program Level Minimum Average Maximum 



Pre Occupational/ 



Family Living 


30 


46 


77 


Intermediate II 


30 


44 


65 


Intermediate I 


30 


43 


78 


Primary II 


30 


45 


70 


Primary I 


45 


47 


51 


Reinforcement and 


30 


44 


62 



Adaptive Behavior 



Enrollment and attendance of Program pupils, 
by Program Level, is summarized here as average 

numbers of days and as rate of attendance: 

Average ' Average Average 

Days Days . Attendance 

Program Level Enrollment Attendance Rate 

Pre Occupaticnal/ 

Fanily Living 160 151 94^. 

Ini.erirediate II 154 134 B7?, 

Intermediate I 164 144 881^ 

Primary II 161 138 86% 

Primary I 173 159 92% 

Reinforcement and 167 152 91% 
Adaptive Behavior 
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The following factors summarize the extent to which the 
overall structure of Program Performance Objectives, as de- 
veloped for the first project year, was found to reflect the 
TMR instructional program. 

4 The Language objectives generally were the most difficult 
for pupils to master and were placed too high to serve 
as realistic terminal objectives for each Program Level. 
The Social Adequacy objectives were the easiest to master 
and, with some exceptions, were reasonably placed as 
terminal objectives for each Program Level. The Vocational 
Readiness objectives fell between the othei two Program 
Areas in terms of difficulty. 

• In terms of Program Levels, Primary I objectives generally 
had the highest achievement rates and were set somewhat 
low to serve as teriainal Program Level objectives. 

Pre Occupational /Fami ly Living objectives were the most 
difficult and generally were set much too high relative 
to pupil ability. The other Program Levels (Primary IT, 
Intermediate I and Intermediate II) fell between these 
two extremes, in increasing order of difficulty. 

• An analysis using the point biserial correlation coefficient 
revealed a generally strong relationship among objectives 
but also indicated, for further analysis, those objectives 
which did not correlate strongly with the overall struc- 
ture. 
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• Reliability coefficients (calculated by the Kuder- 
Richardson tpchnique) showed very high reliability levels 
for the objective structures at each Program Level. 

• The estimation of numbers of objectives gained between 
the first and last test sessions shows that even at the 
lower Program Levels pupils gained relatively few ob- 
jectives on the average, and that the gains were more 
or less consistent across levels. Therefore, the 
higher the Program Level, the larger was the proportion 
of year-end achieved objectives actually gained during 
the period between the first and last testing sessions. 

• To measure the effects of demographic and Program factors 
on pupil performance, a "total score" for each pupil 

• was computed. This score represented the total number 
of objectives achieved by the end of the year. 

With this total score as the dependent variable, a step- 
wise regression analysis was rui: for each Program Level 
using the following 12 variables: school, birth year, 
sex, race, most recent I.Q., visual disability, cere- 
bral palsy, hearing disability, other physical disabili- 
ties, days enrolled, days in f.ttendance, teacher. 

The correlation matrices generated by the regression 
analyses revealed a number of significant relationships. 
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Perhaps the most interesting are the relationships 
which do not show strong correlations. The following 
variables revealed very low (posi^tive or negative) 
correlations with pupil's total objective "score": 
school, sex, race, physical disabilities, enrollment, 
attendance and teacher I 

Strong correlations existed between "score" and "year 
within level" anc^ between "score" and "most recent I.Q.". 
Al so , enrollment and attendance were , as expected , highly 
correlated with each other. 

The results of the regression analyses can be summarized 
in terms of a ranking of the variables relative to their 
overall contribution to the multiple "r" coefficients. 
Of the 12 variables available for inclusion, three were 
never utilized (teacher, visual disability, cerebral 
palsy.) The remaining vari~'bles are ranked here in 
descending order of their overall contribution to the 
multiple "r" coefficients: 

• most recent I.Q. 

• year within Program Level 

• enrollment/attendance 

• race 

• sex 

• school 

• "other" physical disabilities (not visual, 

hearing or cerebral palsy) 

• hearing disability 
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An analysis of available cost and performance data 
indicated that it would not be reasonable to carry 
the cost-effectiveness study for the first year beyond 
the "center" level; the "classroom" level data simply 
was not accurate or complete enough. 

The estimated numbers of objectives achieved by Program 

pupils during the first project year are as follows: 

Center Totals 
#170 #201 



Estimated Total Number 

of Objectives Achieved 2325 975 3300 

Number of Pupils 169 58 227 
Estimated Number of 

Objectives Achieved 14 17 15 
per Pupil 

The results of the cost analysis are summarized here 
as followF: 

Cost Category Center Totals 

#170 #201 

• Center/Classroom Level Costs $207,020 v 79 ,700 $286,720 

• Syst.:m/Program Level Costs $ 29,448 $ 10,107 $ 39,555 

• Total Estimated Costs $236,468 $ 89,807 $326,275 

• Total Estimated Costs $ 1,399 $ 1,548 $ 1,437 

per Pupil — 
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The overall results of the cost-effectiveness analysis 
of the first project year are given by the following 
table: 



Center 



Overall 
Program 



# 170 



# 201 



EstiiTiated Average Cost 
per Pupil 



$1,399 



$1,548 



$1,437 



Estimated Average 
Number of Objectives 
Achieved per Pupil 



14 



17 



15 



Estimated Average Cost 
per Objective Achieved 



$ 100 



91 



96 



The teacher and parent surveys taken in June showed 
generally positive attitudes towa.**d the first year of 
the project. 

Positive comments from teachers related to the effective- 
ness of the accountability model in prescriptive planning 
and in creating in the pupils a sense of accomplishment. 
In general, the teachers did not like the testing pro- 
cedures used during the first year and they felt that 
there were some weaknesses in the definition and structure 
of the objectives. 

In general, most parents felt they were adequately 
informed about the TMR Program during the past year, 
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yet they were not familiar with new methods, procedures 
and techniques used in the program. Also, there was 
an indication on the part of many parents that more 
parental involvement was needed. 



IMPLICATIONS AND RECOMMENDATIONS 



The following implications and recommendations are 
based on our evaluation of the first year of the 
Accountability Study. 

• The overall objective for the first year of the 
project, "To develop, implement and audit an 
evaluational design based on systems analysis, which 
will measure the cost-ef fect.rveness of the Duval 
County Program for Trainable Mentally Retarded 
Children and Youth in achieving its stated goals 
in the areas of Language Development, Social Adequacy 
and Vocational Readiness' , was not fully achieved. 
The procedures utilized did not result in a reliable 
estimate of the cost-effectiveness of v-^e program. 
However, considering the limited budget available 
for th«^ first project year, the results reflect a 
highly productive allocation of project resources. 
Despite a number of faulty design assumptions made 
early in the development of the project, a large 
amount of relevant data was generated and analyzed. 
The results are a potentially effective Accountability 
Model and the procedures essential to its implementa- 
tion. 
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• Major weaknesses in the first year's design related 
to 1) a lack of specificity in the pupil perform- 
ance objectives and the procedures for assessment, 
2) inadequate inservice training for Program 
teachers in the application and measurement of 

the objectives, 3) subjective and unnecessarily 
rigid structuring of objectives relative to Program 
Levels and 4) the limitations of our own evaluation 
design. 

• Major achievements of the first project year include 
1 ) the development of a comprehensive picture of 
the objective structure in terms of validity, 
reliability and degree of difficulty of individual 
objectives, 2) the determination of significant 
relationships among pupil and program factors and 
pupil performance, 3) procedures for and collection 
of preliminary cost and performance data, and 4) de- 
velopment of the Accountability Model and the 
procedures for its implementation. 

• Our recommendations for the continuing development 
and implementation of the Accountability Model have 
been discussed with the DCSB Project Director and, 
where budget constraints allowed, incorporated into 
the design of the second year of the project. 
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Major elements of the second project year include 
1) development of an overall organizing model for 
all performance objectives, 2) revision of objectives 
so that they meet realistic requirements of speci- 
ficity, 3) writing of criterion-referenced items 
for each objective, specifying procedures to be 
used in assessing pupil performance, 4) revision 
of assessment procedures so that a pupil is assessed 
on objectives appropriate for his individual skills 
rather than his chronological age, and 5) extensive 
training of teachers in the objective assessment 
of trainable pupil performance. 

• The Accountability Model developed through this 
project has application to most TMR programs . Based 
on the assessment of pupil needs and the application 
of performance objectives to the instructional 
process, the model provides for the effective allo- 
cation of resources and the communication of results 
to pupils, teachers , parents and community. Although 
validation and implementation of the model are 
continuing through the second project year, a de- 
scription of the model and supporting documentation 
can be obtained from the DCSB Project Coordinator. 
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